Reflections on Eight Years of Instrument Creation with Machine Learning

Machine learning (ML) has been used to create mappings for digital musical instruments for over twenty-five years, and numerous ML toolkits have been developed for the NIME community. However, little published work has studied how ML has been used in sustained instrument building and performance practices. This paper examines the experiences of instrument builder and performer Laetitia Sonami, who has been using ML to build and refine her Spring Spyre instrument since 2012. Using Sonami’s current practice as a case study, this paper explores the utility, opportunities, and challenges involved in using ML in practice over many years. This paper also reports the perspective of Rebecca Fiebrink, the creator of the Wekinator ML tool used by Sonami, revealing how her work with Sonami has led to changes to the software and to her teaching. This paper thus contributes a deeper understanding of the value of ML for NIME practitioners, and it can inform design considerations for future ML toolkits as well as NIME pedagogy. Further, it provides new perspectives on familiar NIME conversations about mapping strategies, expressivity, and control, informed by a dedicated practice over many years

Machine Learning, Music and Creativity: An Interview with Rebecca Fiebrink

Rebecca Fiebrink is a Senior Lecturer at Goldsmiths, University of London, where she designs new ways for humans to interact with computers in creative practice. As a computer scientist and musician, much of her work focuses on applications of machine learning to music, addressing research questions such as: ‘How can machine learning algorithms help people to create new musical instruments and interactions?’ and ‘How does machine learning change the type of musical systems that can be created, the creative relationships between people and technology, and the set of people who can create new technologies?’ Much of Fiebrink’s work is also driven by a belief in the importance of inclusion, participation, and accessibility. She frequently uses participatory design processes, and she is currently involved in creating new accessible technologies with people with disabilities, designing inclusive machine learning curricula and tools, and applying participatory design methodologies in the digital humanities. Fie-brink is the developer of the Wekinator: open-source software for real-time interac-tive machine learning, whose current version has been downloaded over 10,000 times. She is the creator of a MOOC titled “Machine Learning for Artists and Musicians.” She was previously an Assistant Professor at Princeton University, where she co-directed the Princeton Laptop Orchestra. She has worked with companies including Microsoft Research, Sun Microsystems Research Labs, Imagine Research, and Smule. She has performed with a variety of musical ensembles playing flute, keyboard, and laptop. She holds a PhD in Computer Science from Princeton University

Data as design tool. How understanding data as a user interface can make end-user design more accessible, efficient, effective, and embodied, while challenging machine learning conventions.

We often assume "data" is something that is collected or measured from a passive source. In machine learning, we talk about "ground truth" data, because we assume the data represents something true and real; we aim to analyse and represent data appropriately, so that it will yield a window through which we can better understand some latent property of the world. In this talk, I will describe an alternative understanding of data, in which data is something that people can actively, subjectively, and playfully manipulate. Applying modelling algorithms to intentionally manipulated data—such as examples of human movements, sounds, or social media feeds— enables everyday people to build new types of real-time interactions, including new musical instruments, sonifications, or games. In these contexts, data becomes an interface through which people communicate embodied practices, design goals, and aesthetic preferences to computers. This interface can allow people to design new real-time systems more efficiently, to explore a design space more fully, and to create systems with a particular “feel,” while also making design accessible for non-programmers

Creative Diversity: Expanding Software for 3D Human Avatar Design

This research designs 3D human avatar generation software for amateur creative users. Currently available software relies on limiting the range of possible bodies that the user is able to create, within the boundaries of normative physicality, in order to simplify interaction for users without 3D modeling skills. Rather than artificially limiting user output, we are creating open source software that expands the range of bodies able to be represented in program, following a user centered design process to implement direct manipulation techniques extrapolated from artistic practice. This paper describes the background context, aims, and current research activities related to creating this software

Creating Latent Spaces for Modern Music Genre Rhythms Using Minimal Training Data

In this paper we present R-VAE, a system designed for the exploration of latent spaces of musical rhythms. Unlike most previous work in rhythm modeling, R-VAE can be trained with small datasets, enabling rapid customization and exploration by individual users. R-VAE employs a data representation that encodes simple and compound meter rhythms. To the best of our knowledge, this is the first time that a network architecture has been used to encode rhythms with these characteristics, which are common in some modern popular music genres

Introduction to the Special Issue on Human-Centered Machine Learning

Machine learning is one of the most important and successful techniques in contemporary computer science. Although it can be applied to myriad problems of human interest, research in machine learning is often framed in an impersonal way, as merely algorithms being applied to model data. However, this viewpoint hides considerable human work of tuning the algorithms, gathering the data, deciding what should be modeled in the first place, and using the outcomes of machine learning in the real world. Examining machine learning from a human-centered perspective includes explicitly recognizing human work, as well as reframing machine learning workflows based on situated human working practices, and exploring the co-adaptation of humans and intelligent systems. A human-centered understanding of machine learning in human contexts can lead not only to more usable machine learning tools, but to new ways of understanding what machine learning is good for and how to make it more useful. This special issue brings together nine papers that present different ways to frame machine learning in a human context. They represent very different application areas (from medicine to audio) and methodologies (including machine learning methods, HCI methods, and hybrids), but they all explore the human contexts in which machine learning is used. This introduction summarizes the papers in this issue and draws out some common themes

The Machine Learning Algorithm as Creative Musical Tool

Machine learning is the capacity of a computational system to learn structures from datasets in order to make prediction in front of newly seen datasets. Such approach offers a significant advantage in music scenarios in which musicians can teach the system to learn an idiosyncratic style, or can break the rules to explore the system capacity in unexpected ways. In this chapter we draw on music, machine learning, and human-computer interaction to elucidate an understanding of machine learning algorithms as creative tools for music and the sonic arts. We motivate a new understanding of learning algorithms as human-computer interfaces. We show that, like other interfaces, learning algorithms can be characterised by the ways their affordances intersect with goals of human users. We also argue that the nature of interaction between users and algorithms impacts the usability and usefulness of those algorithms in profound ways. This human-centred view of machine learning motivates our concluding discussion of what it means to employ machine learning as a creative tool

Toward Supporting End-User Design of Soundscape Sonifications

In this paper, we explore the potential for everyday Twitter users to design and use soundscape sonifications as an alternative, “calm” modality for staying informed of Twitter activity. We first present the results of a survey assessing how 100 Twitter users currently use and change audio notifications. We then present a study in which 9 frequent Twitter users employed two user interfaces—with varying degrees of automation—to design, customize, and use soundscape sonifications of Twitter data. This work suggests that soundscapes have great potential for creating a calm technology for maintaining awareness of Twitter data, and that soundscapes can be useful in helping people without prior experience in sound design think about sound in sophisticated ways and engage meaningfully in sonification design

The Global Metronome: Absolute tempo sync for networked musical performance

At a time in the near future, many computers (including devices such as smart-phones) will have system clocks that are synchronized to a high degree (less than 1 ms of error). This will enable us to coordinate events across unconnected devices with a degree of accuracy that was previously impossible. In particular, high clock synchronization means that we can use these clocks to synchronize tempo between humans or sequencers with little-to-no communication between the devices. To facilitate this low-overhead tempo synchronization, we propose the Global Metronome, which is a simple, computationally cheap method to obtain absolute tempo synchronization. We present experimental results demonstrating the effectiveness of using the Global Metronome and compare the performance to MIDI clock sync, a common synchronization method. Finally, we present an open source implementation of a Global Metronome server using a GPS-connected Raspberry Pi that can be built for under $100

Interacting with neural audio synthesis models through interactive machine learning

Recent advances in neural audio synthesis have made it possible to generate audio signals in real time, enabling the use of applications in musical performance. However, exploring and playing with their high-dimensional spaces remains challenging, as the axes do not necessarily correlate to clear musical labels and may vary from model to model. In this paper, we present a proof-of-concept mechanism for steering latent audio models through interactive machine learning. Our approach involves mapping the human-performance space to the high-dimensional, computer- generated latent space of a neural audio model by utilizing a regressive model learned from a set of demonstrative actions. By implementing this method in ideation, exploration, and sound and music performance we have observed its efficiency, flexibility, and immediacy of control over generative audio processes